FINE-TUNING METHODS

LLMs and Copyright Risks: Benchmarks and Mitigation Approaches (2025.naacl-tutorial)

Challenge:	Recent advances in large language models have sparked interest in creating autonomous agents.
Approach:	They propose a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents.
Outcome:	The proposed framework improves task planning and self-reflective evolution capabilities in language agents.

Copied to clipboard

Denghui Zhang, Zhaozhuo Xu, Weijie Zhao

Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer (2024.eacl-short)

Challenge:	Large Language Models (LLMs) have revolutionized natural language processing, but their widespread use has raised significant copyright concerns.
Approach:	This tutorial will provide an overview of relevant copyright principles and their application to AI and examine specific copyright issues in LLM development and deployment.
Outcome:	The course will provide an overview of relevant copyright principles and their application to AI, followed by an examination of specific copyright issues in LLM development and deployment.

Copied to clipboard

Marinela Parović, Ivan Vulić, Anna Korhonen

Cross-Lingual Transfer

Challenge:	Massively multilingual Transformer-based models (MMTs) can learn representations which have a degree of cross-lingual alignment despite being trained using purely unsupervised objectives.
Approach:	They propose a modular approach to cross-lingual transfer using task arithmetic . they show that modularity can be achieved even with full model fine-tuning .
Outcome:	The proposed approach shows strong performance on multilingual benchmarks encompassing both high-resource and low-resourced languages.

DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer (2023.eacl-main)

Copied to clipboard

Shanu Kumar, Soujanya Abbaraju, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Multimodal Instruction Tuning with Conditional Mixture of LoRA (2024.acl-long)

Challenge:	Zero-shot cross-lingual transfer has been shown to be sub-optimal across low-resource languages due to the skew in resource distribution in languages.
Approach:	They propose to jointly reduce feature incongruity between the source and target language and increase generalization capabilities of pre-trained multilingual transformers.
Outcome:	Empirical results show that the proposed approach outperforms the standard zero-shot fine-tuning method on multiple datasets across all languages using only unlabeled instances in the target language.

Copied to clipboard

Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang

PILLOW: Enhancing Efficient Instruction Fine-tuning via Prompt Matching (2023.emnlp-industry)

Challenge:	Multimodal Large Language Models (MLLMs) have demonstrated proficiency in diverse tasks across different domains.
Approach:	They propose a method that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA.
Outcome:	Experimental results show that MixLoRA outperforms LoRA with the same or higher ranks . MLLMs have demonstrated remarkable proficiency in diverse tasks across domains .

Copied to clipboard

Zhenting Qi, Xiaoyu Tan, Shaojie Shi, Chao Qu, Yinghui Xu, Yuan Qi

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)

Challenge:	Low-Rank Adaptation (LoRA) has been used to adapt Large Language Models to a variety of tasks, but it requires substantial computational resources to perform.
Approach:	They propose a low-rank adaptive learning approach that leverages LoRA's in-context learning capability through prompt matching via reinforcement learning in resource-constrained environments.
Outcome:	The proposed model improves LoRA performance on evaluation metrics and utilises consumer-grade GPU resources.

Copied to clipboard

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson

OpenICL: An Open-Source Framework for In-context Learning (2023.acl-demo)

Challenge:	State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model.
Approach:	They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks.
Outcome:	The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task.

Copied to clipboard

Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, Yu Qiao

Petals: Collaborative Inference and Fine-tuning of Large Models (2023.acl-demo)

Challenge:	In-context Learning (ICL) is a new paradigm for large language model evaluation.
Approach:	They propose an open-source toolkit for ICL and LLM evaluation.
Outcome:	The proposed framework is highly flexible and flexible and can be easily combined with other tools to suit users' needs.

Copied to clipboard

Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel

Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2026.findings-acl)

Challenge:	Recent studies show that pretrained language models can solve practical tasks using more than 100 billion parameters.
Approach:	They propose a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties.
Outcome:	The proposed system outperforms offloading for very large models running on consumer GPUs with 1 step per second, enough for many interactive LLM applications.

Copied to clipboard

Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Li Yabo, Haozhe Ma, Yangqiu Song, Bryan Hooi

Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation (2025.emnlp-industry)

Challenge:	Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions.
Approach:	They propose a prompt injection defense method that suppresses the model's instruction-following tendencies rather than suppressing them.
Outcome:	The proposed method outperforms prompt-engineering-based approaches and fine-tuning methods and reduces the ASR to nearly 0% in some scenarios.

Copied to clipboard

Xujun Peng, Anoop Kumar, Jingyu Wu, Parker Glenn, Daben Liu

Information Retrieval

Challenge:	RAG systems often generate inconsistent outputs for semantically equivalent inputs . this unpredictability undermines the reliability of RAG and poses challenges for adoption in high-stakes or knowledge-sensitive domains such as finance, healthcare, and scientific research.
Approach:	They propose a method that integrates knowledge from specialized models into a single model to improve output consistency.
Outcome:	The proposed model significantly improves output consistency, achieving approximately 47.5% improvement in response similarity over baseline.

E5: Zero-shot Hierarchical Table Analysis using Augmented LLMs via Explain, Extract, Execute, Exhibit and Extrapolate (2024.naacl-long)

Copied to clipboard

Zhehao Zhang, Yan Gao, Jian-Guang Lou

RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Models (2024.findings-acl)

Challenge:	Hierarchical tables with multi-level headers are confusing for models due to their complex structure, implicit semantics, and calculation relationships.
Approach:	They propose a code-augmented framework for zero-shot hierarchical table question answering that combines self-explaining the table's hierarchic structures with external code execution to prevent hallucinations.
Outcome:	The proposed framework outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement.

Copied to clipboard

Saeed Najafi, Alona Fyshe

SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models (2024.findings-naacl)

Challenge:	Pre-trained Language Models (PLMs) can be fine-tuned for downstream text processing tasks.
Approach:	They propose to use paraphrases to enrich the input text of a few-shot model with a Maximum-Marginal Likelihood objective to improve performance.
Outcome:	The proposed methods improve performance beyond what can be achieved with parameter-efficient fine-tuning alone.

Copied to clipboard

Hossein Hajipour, Ning Yu, Cristian-Alexandru Staicu, Mario Fritz

Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (2020.coling-main)

Challenge:	Large datasets are increasingly available for pre-training source code models, but obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources.
Approach:	They propose a systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and investigates model behavior under different fine-tuning methodologies.
Outcome:	The proposed approach simulates various OOD scenarios along different dimensions of source code data properties and exposes multiple failure modes attributed to OOD generalization issues.

Copied to clipboard

Daniel Grießhaber, Johannes Maucher, Ngoc Thang Vu

Efficient Few-Shot Fine-Tuning for Opinion Summarization (2022.findings-naacl)

Challenge:	Recent work has explored the suitability of pre-trained language models in low resource settings with less than 1,000 training data points.
Approach:	They propose to use pool-based active learning to speed up training while keeping the cost of labeling new data constant.
Outcome:	The proposed model can be fine-tuned to optimize for low-resource settings while keeping the cost of labeling constant.

Copied to clipboard

Arthur Brazinskas, Ramesh Nallapati, Mohit Bansal, Markus Dreyer

An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models (2022.emnlp-main)

Challenge:	Abstractive summarization models are typically pre-trained on large amounts of generic texts . large annotated datasets of reviews paired with reference summaries are not available .
Approach:	They propose a few-shot method which uses adapters to store in-domain knowledge . they pre-train adapters on unannotated customer reviews and fine-tune them on annotated datasets .
Outcome:	The proposed method can store in-domain knowledge and improves on large annotated reviews . it improves coherence and redundancies on the Amazon and Yelp datasets .

Copied to clipboard

Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, Taylor Berg-Kirkpatrick

Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)

Challenge:	Large language models are shown to present privacy risks through memorization of training data, but little attention has been given to the fine-tuning phase.
Approach:	They empirically study memorization of fine-tuning methods using membership inference and extraction attacks and show that fine-timing the head of the model has the highest susceptibility to attacks.
Outcome:	The proposed methods have the highest memorization risk, whereas the smaller adapters are less vulnerable to known extraction attacks.

Copied to clipboard

Hamish Ivison, Matthew Peters

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision (2021.emnlp-main)

Challenge:	Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously.
Approach:	They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance.
Outcome:	The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels.

Copied to clipboard

Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning (2024.emnlp-main)

Challenge:	Recent state-of-the-art (SOTA) effective neural network methods have been used in Chinese word segmentation (CWS) However, the robustness of the previous neural methods is limited by the large-scale annotated corpus.
Approach:	They propose a self-supervised Chinese word segmentation approach with a straightforward and effective architecture.
Outcome:	The proposed approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness.

Copied to clipboard

Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang

The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training (2026.findings-acl)

Challenge:	Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing.
Approach:	They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning.
Outcome:	The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability.

Copied to clipboard

Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes, Yang Zhang

LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)

Challenge:	Misaligned large language models can magnify harm by exploiting them to undermine safety . et al., 2022b; Bai e.t., 2023): misalignment, realignment and model-specific resistance are important .
Approach:	They evaluate four methods to identify a mechanism asymmetry between attack and defense . they find that ORPO is most effective for misalignment, but DPO excels in realignment .
Outcome:	The proposed methods show a mechanism asymmetry between attack and defense . the proposed methods excel in realignment, but at the expense of model utility .

Copied to clipboard

Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao

LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation (2024.findings-emnlp)

Challenge:	LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings.
Approach:	They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning.
Outcome:	The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks.

Copied to clipboard

Yinqiao Li, Linqi Song, Hanxu Hou

ROSE: Robust Selective Fine-tuning for Pre-trained Language Models (2022.emnlp-main)

Challenge:	Recent methods for fine-tuning large language models have shown great improvements on a wide range of NLP tasks.
Approach:	They propose to introduce a non-linear transformation to improve performance of adapters by introducing a low-rank adaptation to fit the accumulated weight updates.
Outcome:	The proposed method outperforms a baseline on SAMSum and 20 Newsgroups tasks and even improves the classification task by 1.95 points when a lower rank is applied.

Copied to clipboard

Lan Jiang, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Rui Jiang

Robustness in NLP

Challenge:	Recent studies have highlighted the lack of adversarial robustness in pre-trained models.
Approach:	They propose a fine-tuning approach that conducts selective updates when adapting pre-trained models to downstream tasks.
Outcome:	The proposed approach improves adversarial robustness on downstream tasks . it eliminates spurious updates, leading to flatter and wider optima than the conventional method .

CIE: Controlling Language Model Text Generations Using Continuous Signals (2025.emnlp-main)

Copied to clipboard

Vinay Samuel, Harshita Diddee, Yiming Zhang, Daphne Ippolito

MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling (2023.acl-long)

Challenge:	Existing methods to control language models with intent are brittle and hard to scale.
Approach:	They propose to use a set of LMs to fine-tune to expect a control vector that is interpolated between a "low" and a 'high' token embedding.
Outcome:	The proposed method can be finetuned to expect a control vector that is interpolated between a “low” and a ‘high” token embedding.

Copied to clipboard

Yu Song, Santiago Miret, Bang Liu

Soft Prompting for Unlearning in Large Language Models (2025.naacl-long)

Challenge:	Using publicly available materials science text data, we construct a benchmark for evaluating the performance of natural language processing (NLP) models on materials science texts.
Approach:	They propose a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text.
Outcome:	The proposed model outperforms BERT-based models on scientific text and a model pretrained on materials science journals.

Copied to clipboard

Karuna Bhaila, Minh-Hao Van, Xintao Wu

Exploring the Impact of Personality Traits on LLM Toxicity and Bias (2025.emnlp-main)

Challenge:	Existing ethical and safety considerations for large language models are important for deployment . however, some ethical concerns have been raised due to the presence of private, sensitive, or harmful information in the training data.
Approach:	They propose a framework that learns prompt tokens that are prepended to a query to induce unlearning in LLMs.
Outcome:	The proposed method improves the trade-off between utility and forgetting for text classification and question-answering.

Copied to clipboard

Shuo Wang, Renhao Li, Xi Chen, Yulin Yuan, Min Yang, Derek F. Wong

Sentiment Analysis

Challenge:	anthropomorphic LLMs are being developed to serve diversified roles, but content safety concerns remain regarding their toxicity and toxicity.
Approach:	They propose to assign personality traits to large language models (LLMs) to reduce toxic language and social biases in their outputs by using the widely accepted HEXACO personality framework developed in social psychology.
Outcome:	The proposed model is able to perform on three toxic and bias benchmarks and shows that assigning personality traits reduces bias and toxicity similar to humans’ correlations between personality traits and toxic behaviors.

Extractive Summarization via ChatGPT for Faithful Summary Generation (2023.findings-emnlp)

Copied to clipboard

Haopeng Zhang, Xiao Liu, Jiawei Zhang

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Abstractive summarization methods struggle with generating ungrammatical or even nonfactual contents.
Approach:	They evaluate ChatGPT's performance on extractive summarization and compare it with traditional fine-tuning methods on benchmark datasets.
Outcome:	The proposed pipeline performs better than abstractive methods on summary faithfulness and in-context learning.

Copied to clipboard

Zeqi Tan, Yongliang Shen, Xiaoxia Cheng, Chang Zong, Wenqi Zhang, Jian Shao, Weiming Lu, Yueting Zhuang

Green & Sustainable NLP

Challenge:	Large language models (LLMs) have shown remarkable performance, but their training costs are exorbitant.
Approach:	They propose a parameter-efficient method for exploring optimal solutions within latent space by using latent units to extract input representations from LLMs.
Outcome:	The proposed method improves performance on a range of natural language processing tasks.

RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training (2023.findings-emnlp)

Copied to clipboard

Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, Madian Khabsa

Robustness in NLP

Challenge:	Several perspectives of robustness for pre-trained language models have been studied independently, but lacking a unified consideration in multiple perspectives.
Approach:	They propose a technique to enhance the multi-perspective robustness of LMs by introducing adversarial perturbation while the model parameters are selectively updated upon their relative importance.
Outcome:	The proposed technique improves the robustness of LMs by incorporating four perspectives on model robustness.

ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context (2024.findings-emnlp)

Copied to clipboard

Zirui Wu, Yansong Feng

Learning to Transfer Prompts for Text Generation (2022.naacl-main)

Challenge:	Tables are a crucial tool for organizing and presenting information in various domains.
Approach:	They propose a Plan-then-Reason framework to answer different types of user queries over tables with sentence context.
Outcome:	The proposed framework outperforms existing frameworks without self-consistency while using less API calls and in-context demonstrations.

Copied to clipboard

Junyi Li, Tianyi Tang, Jian-Yun Nie, Ji-Rong Wen, Xin Zhao

The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding (N19-1)

Challenge:	Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning.
Approach:	They propose a prompt-based method that learns source prompts and transfers them as target prompts to perform target generation tasks.
Outcome:	The proposed method can be used to perform text generation tasks in a transferable setting.

Copied to clipboard

Yiqun Yao, Jiaming Xu, Bo Xu

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)

Challenge:	Visual Dialog is a multi-modal task that requires a model to participate in a dialog grounded on an image and generate correct, human-like responses.
Approach:	They propose a framework for effective and robust auxiliary training of visual dialog systems using multi-modal encoding.
Outcome:	The proposed framework outperforms supervised learning baselines and fine-tuning methods on most metrics of VisDial v0.5/v0.9 generative tasks.

Copied to clipboard

Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang

GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent (2025.acl-long)

Challenge:	Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference.
Approach:	They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes.
Outcome:	The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training.

Copied to clipboard

Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, Liqiang Nie

Low-Resource NLP

Challenge:	GUI automation is a key challenge in dynamic environments.
Approach:	They propose a training-free GUI agent that integrates two mechanisms to explore trajectories in GUIs.
Outcome:	The proposed GUI-explorer shows significant improvements over existing agents.

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)

Copied to clipboard

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee

Towards More Efficient Post-training via Fourier Domain Adapter Framework (2025.findings-emnlp)

Challenge:	Large language models (LLMs) have shown unprecedented performance across various tasks.
Approach:	They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks .
Outcome:	The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data.

Copied to clipboard

Yijia Fan, Jusheng Zhang, Keze Wang

Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration (2022.acl-long)

Challenge:	FDA reparameterizes the core projection operation of the adapter module directly in the Fourier domain.
Approach:	They propose a framework that reparameterizes the core projection operation of the adapter module directly in the Fourier domain.
Outcome:	The proposed framework outperforms existing parameter-efficient fine-tuning methods on GLUE, E2E NLG, and instruction tuning benchmarks.

Copied to clipboard

Xiwen Liang, Fengda Zhu, Li Lingling, Hang Xu, Xiaodan Liang

Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization (2023.findings-emnlp)

Challenge:	Existing methods of fine-tuning vision-language navigation models require extra human-labeled data and lack self-exploration capabilities in environments.
Approach:	They propose a method that can self-explore environments without human labeling . they use a large-scale cross-modal pretrained model to build an in-domain dataset .
Outcome:	The proposed model can self-explore environments without human labeling without human supervision and generates structured instructions without human intervention.

Copied to clipboard

Shoujie Tong, Heming Xia, Damai Dai, Runxin Xu, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui

Harnessing Pre-Trained Neural Networks with Rules for Formality Style Transfer (D19-1)

Challenge:	Pretrained language models can be fine-tuned on limited training data, which can overfit and thus diminish performance.
Approach:	They propose a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout.
Outcome:	The proposed method outperforms existing methods on the GLUE benchmark and exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.

Copied to clipboard

Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, Wenhan Chao

Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning (2022.coling-1)

Challenge:	Existing studies normalize informal sentences with rules, but they introduce noise if we use them in a naive way.
Approach:	They propose to harness rules into a state-of-the-art neural network that is typically pretrained on massive corpora.
Outcome:	The proposed method can be used to generate a state-of-the-art on a small dataset.

Copied to clipboard

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, Dietrich Klakow

Multilinguality

Challenge:	Multilingual pre-trained language models have shown impressive performance on several downstream tasks for both high-resourced and low-resource languages.
Approach:	They propose to apply multilingual adaptive fine-tuning to 17 most-resourced African languages and three other high-resource languages to encourage cross-lingual transfer learning.
Outcome:	The proposed approach is competitive to LAFT on individual languages while requiring significantly less disk space.

Embedding Hallucination for Few-shot Language Fine-tuning (2022.naacl-main)

Copied to clipboard

Yiren Jian, Chongyang Gao, Soroush Vosoughi

GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Models (2025.findings-naacl)

Challenge:	Existing methods for fine-tuning pre-trained language models can cause severe over-fitting.
Approach:	They propose an Embedding Hallucination method which generates auxiliary embedding-label pairs to expand the fine-tuning dataset.
Outcome:	The proposed method outperforms current fine-tuning methods in a wide range of language tasks.

Copied to clipboard

Aditya Sharma, Aman Dalmia, Mehran Kazemi, Amal Zouaq, Christopher Pal

AdaDHP: Fine-Grained Fine-Tuning via Dual Hadamard Product and Adaptive Parameter Selection (2025.acl-long)

Challenge:	Various vision-language models (VLMs) have made significant progress in multimodal tasks, but they still struggle with geometry problems.
Approach:	They propose a vision-language model that leverages modular code-finetuning to generate and execute code using a predefined geometry function library.
Outcome:	The proposed model improves geometric reasoning abilities by 16% on a GeomVerse dataset compared to other methods.

Copied to clipboard

Han Liu, Changya Li, Xiaotong Zhang, Feng Zhang, Fenglong Ma, Wei Wang, Hong Yu

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025.naacl-long)

Challenge:	Increasing number of parameters can be challenging under resource-constrained environments.
Approach:	They propose a parameter-efficient fine-tuning method with fewer parameters and finer granularity that can adaptively select important parameters for each task.
Outcome:	The proposed method can fine-tune important parameters for each task, while maintaining the same weights.

Copied to clipboard

Hyegang Son, Yonglak Son, Changhoon Kim, Young Geun Kim

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)

Challenge:	Existing methods for fine-tuning pre-trained models impose substantial resource usage.
Approach:	They propose a parameter-efficient fine-tuning method that freezes adapters early to reduce resource usage while maintaining performance.
Outcome:	The proposed method reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% while maintaining performance.

Copied to clipboard

Zhen Zhang, Jialu Wang, Xin Wang

Multilinguality

Challenge:	Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages.
Approach:	They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities.
Outcome:	The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters.

Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference (2023.acl-long)

Copied to clipboard

Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen

Soft Language Prompts for Language Transfer (2025.naacl-long)

Challenge:	Existing studies attribute catastrophic forgetting to fine-tuning, and they retain pre-trained knowledge indiscriminately without identifying what knowledge is transferable.
Approach:	They propose a unified objective for fine-tuning to retrieve the causality back from pre-trained data and use it to mitigate negative transfer while preserving knowledge.
Outcome:	The proposed method outperforms state-of-the-art fine-tuning methods on commonsense QA datasets and can be implemented as a plug-in module to inflate the performance of existing QA models.

Copied to clipboard

Ivan Vykopal, Simon Ostermann, Marian Simko

LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs? (2025.naacl-long)

Challenge:	Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing.
Approach:	They propose to combine language-specific adapters and soft prompts to enhance cross-lingual transfer by parameter-efficient fine-tuning methods.
Outcome:	The proposed methods outperform language adapters and soft prompts in 16 languages and 10 low-resource languages.

Copied to clipboard

Jan Cegin, Jakub Simko, Peter Brusilovsky

Information Extraction & Text Mining

Challenge:	Recent studies have compared LLM-based augmentations with established methods, but the results are contradictory.
Approach:	They compare the performance of LLM-based augmentation methods with established ones . they found that LLMs are worthy of deployment only when very small number of seeds is used .
Outcome:	The proposed methods are worthy of deployment only when very small number of seeds is used.

RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation (2026.findings-acl)

Copied to clipboard

Kaiqu Liang, Haimin Hu, Ryan Liu, Thomas L. Griffiths, Jaime Fernández Fisac

BeamLoRA: Beam-Constraint Low-Rank Adaptation (2025.acl-long)

Challenge:	Reinforcement Learning from Hindsight Simulation (RLHF) can cause severe misalignment in generative AI, but it is not a universal method for fine-tuning large language models.
Approach:	They propose a method that uses evaluator feedback to decouple alignment signal from potentially compromised predictions.
Outcome:	The proposed method significantly outperforms RLHF in comparisons with baselines and human evaluations.

Copied to clipboard

Naibin Gu, Zhenyu Zhang, Xiyu Liu, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang

Reasoning

Challenge:	Low-Rank Adaptation (LoRA) is one of the most efficient parameter-efficient fine-tuning methods.
Approach:	They propose to conceptualize each LoRA module as a beam where each rank corresponds to a potential sub-solution.
Outcome:	The proposed method improves performance on three base models and 12 datasets.

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning (2025.coling-main)

Copied to clipboard

Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang

Fair Federated Learning with Biased Vision-Language Models (2024.findings-acl)

Challenge:	Low-rank adaptation and its variants have been popular due to their ability to avoid excessive inference costs.
Approach:	They propose a low-rank adaptation method that enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight.
Outcome:	The proposed method outperforms LoRA and other fine-tuning methods across tasks with less trainable parameters.

Copied to clipboard

Huimin Zeng, Zhenrui Yue, Yang Zhang, Lanyu Shang, Dong Wang

Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning (2025.coling-main)

Challenge:	Existing literature ignores the inherent group unfairness within CLIP and its ethical implications on FL applications.
Approach:	They propose a fairness-aware adaptation framework for CLIP in federated learning . they propose to leverage biased pre-trained VLMs to build fair FL frameworks .
Outcome:	The proposed framework addresses unique bias in FL, triggered by data heterogeneity . it trains a fair FL model with fairness-aware deep visual prompting (DVP) Extensive results on human face attribute recognition (FAR) applications show it outperforms state-of-the-art FL models .

Copied to clipboard

Jing Bi, Yuting Wu, Weiwei Xing, Zhenjie Wei

A Two-Stage Masked LM Method for Term Set Expansion (2020.acl-main)

Challenge:	Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks.
Approach:	They propose a new reasoning strategy Solution Guidance (SG) and a plug-and-play training paradigm Solution-Guidance Fine-Tuning (SGFT) which focuses on problem understanding and decomposition at the semantic and logical levels, rather than specific computations.
Outcome:	The proposed reasoning strategy Solution Guidance (SG) and plug-and-play training paradigm Solution-Guidance Fine-Tuning (SGFT) improves the reasoning capabilities of small language models on various reasoning tasks.

Copied to clipboard

Guy Kushilevitz, Shaul Markovitch, Yoav Goldberg

Let’s Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model (2025.coling-main)

Challenge:	Existing methods for Term Set Expansion are either distributional or pattern-based . Term set expansion is a task of expanding a small seed set of example terms into a larger set of terms that belong to the same semantic category.
Approach:	They propose a method which uses neural masked language models to expand a small seed set of terms into a larger set of semantic terms.
Outcome:	The proposed method outperforms state-of-the-art methods due to the small seed set size . it uses neural masked language models to query large, pre-trained mlms .

Copied to clipboard

Haoyun Xu, Runzhe Zhan, Yingpeng Ma, Derek F. Wong, Lidia S. Chao

PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation (2023.findings-acl)

Challenge:	Large Language Models (LLMs) are composed of neurons that exhibit diverse behaviors and roles.
Approach:	They propose a novel approach that refines the granularity of parameter training down to the individual neuron, enabling a more parameter-efficient fine-tuning model.
Outcome:	The proposed approach exceeds the performance of full-parameter fine-tuning and PEFT and provides insights into the analysis of neurons.

Copied to clipboard

Yixin Wan, Kuan-Hao Huang, Kai-Wei Chang

Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning (2024.acl-long)

Challenge:	Existing fine-tuning methods for this task are costly and require updating the parameters of the entire model to adapt to the newly included syntax information.
Approach:	They propose a method to instruct model’s encoder prefix to capture syntax-related knowledge by direct initiation and indirect optimization.
Outcome:	The proposed methods are 10 times more efficient and learnable than existing methods.

Copied to clipboard

Tuc Nguyen, Thai Le

Green & Sustainable NLP

Challenge:	Existing methods for fine-tuning Pretrained Language Models (PLMs) focus on training multiple adapters and continuously adding more adapters for incoming new tasks, making learning among the tasks unequal.
Approach:	They propose to mix only a select sub-set of adapters during inference via model weight averaging to optimize performance on unseen, in-domain examples.
Outcome:	The proposed methods can integrate a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but only a select sub-set of adapters during inference on unseen, in-domain examples.

Transparent and Coherent Procedural Mistake Detection (2025.emnlp-main)

Copied to clipboard

Shane Storks, Itamar Bar-Yossef, Yayuan Li, Zheyuan Zhang, Jason J Corso, Joyce Chai

Challenge:	Procedural mistake detection (PMD) is a problem of classifying whether a human user has successfully executed a task.
Approach:	They extend PMD to require generating visual self-dialog rationales to inform decisions . they leverage a natural language inference model to formulate two automated metrics for coherence of generated rationale.
Outcome:	The proposed model improves on a reframed task with a natural language inference model and a multi-faceted metrics visualization of common outcomes.

HMCL: Task-Optimal Text Representation Adaptation through Hierarchical Contrastive Learning (2025.findings-emnlp)

Copied to clipboard

Zhenyi Wang, Yapeng Jia, Haiyan Ning, Peng Wang, Dan Wang, Yitao Cao

MultiLogicNMR(er): A Benchmark and Neural-Symbolic Framework for Non-monotonic Reasoning with Multiple Extensions (2025.emnlp-main)

Challenge:	Hierarchical Multilevel Contrastive Learning (HMCL) improves text representation for general large language models.
Approach:	a new contrastive learning framework is developed to improve general large language models . HMCL integrates 3-level semantic differentiation and unifies contrastive and pair classification into a strategy .
Outcome:	HMCL outperforms unsupervised methods and supervised fine-tuning approaches in multi-domain and multilingual benchmarks.

Copied to clipboard

Yeliang Xiu, Yongmei Liu

Reasoning

Challenge:	Non-monotonic reasoning is widely used in daily life and legal reasoning.
Approach:	They propose a neural-symbolic framework for multi-extension NMR and propose to build two variants with more extensions or text diversity.
Outcome:	The proposed framework outperforms prompt-based methods and outperformed some fine-tuning methods.

Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation (2024.emnlp-main)

Copied to clipboard

Matthew Raffel, Victor Agostinelli, Lizhong Chen

Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Current fine-tuning methods to adapt LLMs for simultaneous translation suffer from several issues, such as unnecessarily expanded training sets, increased prompt sizes, or restriction to a single decision policy.
Approach:	They propose a new paradigm for fine-tuning large language models for simultaneous translation using an attention mask approach.
Outcome:	The proposed model improves translation quality compared to state-of-the-art models on five language pairs while reducing the computational cost.

Copied to clipboard

Po-Heng Chen, Yun-Nung Chen

On the Relation Between Fine-Tuning, Topological Properties, and Task Performance in Sense-Enhanced Embeddings (2025.acl-long)

Challenge:	Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks.
Approach:	They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks .
Outcome:	The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters.

Copied to clipboard

Deniz Ekin Yavas, Timothée Bernard, Benoit Crabbé, Laura Kallmeyer

Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)

Challenge:	Enhanced word embeddings do not align well with word senses, resulting in poor performance on word sense identification tasks.
Approach:	They propose to use two methods to fine-tune embeddings to identify the topological properties that contribute to sense-enhanced embeddables.
Outcome:	The proposed methods improve the embeddings’ ability to capture nuanced semantic distinctions while reducing their expressiveness.

Copied to clipboard

Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang

WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development (2025.acl-long)

Challenge:	Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass.
Approach:	They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens.
Outcome:	The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost .

Copied to clipboard

Shanchao Liang, Nan Jiang, Shangshu Qian, Lin Tan

Relation Classification via Bidirectional Prompt Learning with Data Augmentation by Large Language Model (2024.lrec-main)

Challenge:	Large Language Models (LLMs) have shown promise in generating source code, but two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML’s hierarchical structure for LLMs; and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code.
Approach:	They propose a structure-aware attention mechanism that uses a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code.
Outcome:	The proposed model outperforms existing methods on the WebSight-Test and Design2Code benchmarks.

Copied to clipboard

Yizhi Jiang, Jinlong Li, Huanhuan Chen

Low-Resource NLP

Challenge:	Recent studies investigate Relation Extraction task from two different aspects.
Approach:	They propose to use Large Language Model (LLM) to do data augmentation and propose a bidirectional prompt template for prompt learning.
Outcome:	The proposed model outperforms the state-of-the-art on four datasets and outperformed existing methods on TACREV, RETACRED and Semeval.

Self-Training Elicits Concise Reasoning in Large Language Models (2025.findings-acl)

Copied to clipboard

Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, Se-Young Yun

Temporal Sampling for Forgotten Reasoning in LLMs (2026.acl-long)

Challenge:	Chain-of-thought reasoning has enabled large language models to use additional computation through intermediate tokens to solve complex tasks, but current models often generate more tokens than necessary to accomplish the task, incurring extraneous inference costs.
Approach:	They propose to fine-tune models with self-generated concise reasoning paths obtained by best-of-N sampling and few-shot conditioning in task-specific settings to elicit concise reasoning.
Outcome:	The proposed method reduces output tokens by 30% on GSM8K and MATH while maintaining average accuracy.

Copied to clipboard

Yuetai Li, Zhangchen Xu, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Xiang Yue, Radha Poovendran

CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models (2025.findings-acl)

Challenge:	a new metric measures the percentage of questions that were answered incorrectly during fine-tuning .
Approach:	They propose a decoding strategy that draws outputs from multiple checkpoints along the training trajectory.
Outcome:	The proposed method improves reasoning performance and consistency across benchmarks.

Copied to clipboard

Xintong Wang, Jingheng Pan, Liang Ding, Longyue Wang, Longqin Jiang, Xingshan Li, Chris Biemann

BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) achieve excellent performance through pretraining on extensive data.
Approach:	They propose an efficient selective layer intervention based on parameter-efficient fine-tuning methods to select the optimal steering layer to modulate LLM semantics.
Outcome:	The proposed approach is based on a model-agnostic framework and is safe to deploy.

Copied to clipboard

Tianyuan Huang, Zepeng Zhu, Hangdi Xing, Zirui Shao, Zhi Yu, Chaoxiong Yang, Jiaxian He, Xiaozhong Liu, Jiajun Bu

A Parameter-Efficient and Fine-Grained Prompt Learning for Vision-Language Models (2025.acl-long)

Challenge:	Existing Braille research focuses on isolated tasks while mixed-content Braille tasks face data scarcity and ambiguities.
Approach:	They propose a syntax tree-based augmentation method tailored for Braille data.
Outcome:	The proposed method improves Braille translation, formula-to-Braille conversion, and mixed-text translation.

Copied to clipboard

Yongbin Guo, Shuzhen Li, Zhulin Liu, Tong Zhang, C.L.Philip Chen

Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models (2025.acl-long)

Challenge:	Current vision-language models extract semantic information from large-scale cross-modal associations, limiting performance and efficiency.
Approach:	They propose a detail-oriented prompt learning method to implement fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters.
Outcome:	The proposed method implements fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters.

Copied to clipboard

Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao

3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction (2025.emnlp-main)

Challenge:	Existing methods for fine-tuning large language models require expensive training on consumer-grade hardwares.
Approach:	They propose a sparsity-based approach that introduces trainable sparse adaptations to the weight matrices in the model and offers greater flexibility in selecting fine-tuned parameters.
Outcome:	The proposed method outperforms other methods for a simple yet effective baseline for nLP tasks while sacrificing performance.

Copied to clipboard

Longxuan Ma, Xiao Wu, Yuxin Huang, Shengxiang Gao, Zhengtao Yu

The Dominance of Text Space: Unveiling the Asymmetric Nature of Cross-Modal Alignment in Large Language Models (2026.acl-long)

Challenge:	Existing approaches to improve sentence representations lack fine-grained guidance on reducing redundant information.
Approach:	They propose a method that dynamically identifies redundant information from a dimensional perspective and trains the SRL model to redistribute semantics on different dimensions.
Outcome:	The proposed method improves sentence representations on seven semantic text similarity benchmarks.

Copied to clipboard

Linqing Chen, Hanmeng Zhong, Wentao Wu, Peng Zhou